Boundary-based MWE segmentation with text partitioning

In this article we present a novel algorithm for the task of comprehensively segmenting texts into MWEs. With the basis for this algorithm (referred to as text partitioning) being recently developed, these results constitute its first performance-evaluated application to a natural language processing task. A differentiating feature of this single-parameter model is its focus on gap (i.e., punctuation) crossings as features for MWE identification, which uses substantially more information in training than is present in dictionaries. We show that this algorithm is capable of achieving high levels of precision and recall, using only type-level information, and then extend it to include part-of-speech tags to increase its performance to state-of-the-art levels, despite a simple decision criterion and general feature space (which makes the method directly applicable to other languages). Since the existence of comprehensive MWE annotations are what drive this segmentation algorithm, these results support their continued production. In addition, we have updated and extended the strength-averaging evaluation scheme, allowing for a more accurate and fine-grained understanding of model performance, and leading us to affirm the differences in nature and identifiability between weaklyand strongly-linked MWEs, quantitatively.